Doped Mott insulators in (111) bilayers of perovskite transition- metal oxides with the 

strong spin-orbit coupling 
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We study the electronic properties of Mott insulators realized in (111) bilayers of perovskite 
transition-metal oxides. The low-energy effective Hamiltonians for such Mott insulators are derived 
in the presence of the strong spin-orbit coupling. These models are characterized by the antiferro- 
magnetic Heisenberg interaction and the anisotropic interaction whose form depends on the d orbital 
occupancy. From exact diagonalization analyses on finite clusters, the ground state phase diagrams 
are derived, including a Kitaev spin liquid phase in a narrow parameter regime for t2g systems. 
Slave-boson mean-field analyses indicate the possibility of novel superconducting states induced 
by carrier doping into the Mott-insulating parent systems, suggesting the present model systems 
as unique play grounds for studying correlation- induced novel phenomena. Possible experimental 
realizations are also discussed. 

PACS numbers: 71.27.+a, 74.20.-z 



Competition and cooperation between Mott physics 
and the rclativistic spin-orbit coupling (SOC) have 
become a central issue in condensed matter physics. 
As these two effects become comparable, 4c? and 5d 
transition-metal oxides (TMOs) could be ideal platforms 
to explore novel phenomena originating from such in- 
teractions. This brought considerable attention to irid- 
ium oxidesji"— Of particular interest is A2lr03 (A=Li or 
Na) where Ir ions form the honeycomb lattice. Density- 
functional-theory calculations for Na2lr03 predicted the 
quantum spin Hall effecti^ Alternatively, with strong 
correlation effects, the low-energy properties of A2lr03 
could be described by a combination of pseudodipolar 
interaction and Heisenberg interaction;^ called Kitaev- 
Heisenberg (KH) model,— which is a candidate for realiz- 
ing Z2 quantum spin liquid (SL) states. However, later 
experimental measurements confirmed a magnetic long- 
range order-IiS in Na2lr03 possiblly because of longer- 
range magnetic couplingSi^iiS The effect of carrier doping 
into the Kitaev-Heisenberg model was also studied j^^i^^ 

Interacting electron models on a honeycomb lattice 
have long been theoretical targets for realizing novel phe- 
nomena such as the quantum Hall effect without Landau 
levels^ and the spin Hall effect with the SOC." The spin 
Hall effect could also be generated by correlations with- 
out the SOCji^ Yet, experimental demonstrations for 
such correlation-induced phenomena remains to be done. 
Recently, artificial bilayers of perovskite TMOs grown 
along the [111] crystallographic axis, where transition- 
metal ions form the buckled honeycomb lattice (Fig. [T]) , 
were proposed as new platforms to explore a variety of 
quantum Hall effects f^^"— This proposal was motivated 
by the recent development in synthesizing artificial het- 
erostructures of TMOs.^^ TMO heterostructures have 
great tunability over fundamental physical parameters, 
including the local Coulomb repulsion, SOC, and carrier 
concentration. However, the effect of correlations to pos- 
sible novel phenomena near Mott insulating states with 
the strong SOC remains to be explored. 

Here, we address the correlation effects in TMO (111) 



FIG. 1: (Color online) Buckled honeycomb lattice realized in 
a (111) bilayer of the cubic lattice. x,y and z in (a) show the 
cubic axes and the spin components in the Kitaev interaction. 



bilayers with the strong SOC. Specifically, we consider t\g 
systems and e^'^ systems for which the low-energy elec- 
tronic properties could be described in terms of 5* = 1/2 
isospins.— We derive the effective Hamiltonians for such 
Mott insulators and analyze them numerically and ana- 
lytically. The effective Hamiltonian for has the form 
of the KH model^ but the SL was found to exist only in a 
small parameter regime. On the other hand, the effective 
Hamiltonian for e^'^ has the Ising-type anisotropy, thus 
the SL is absent. The effect of carrier doping is analyzed 
using slave-boson mean-field methods including ansatze 
which reduce to exact solutions at limiting cases of zero 
doping. It is shown that carrier doping makes the physics 
of our model systems more interesting by inducing un- 
conventional superconducting states, most-likely d + id 
paring which breaks time-reversal symmetry. 

Effective models: We start from a multiband Hubbard 
model with t2g orbitals or Eg orbitals. In both cases, 
only the nearest-neighbor (NN) hoppings are considered, 
and the hopping amplitude is derived from the Slater- 
Koster formula^^ with oxygen p orbitals located between 
the neighboring two d orbitals. The explicit forms of the 
multiband Hubbard models are given in Ref. 

The low-energy effective Hamiltonian for <2g systems 
is derived from the second-order perturbation processes 
with respect to the transfer terms and by projecting the 
superexchange-type interactions onto the isospin states 



2 



for J|^^ =±l/2i22 

\-^eff = 0-) = -^{i\a,-<7) - a\b,-a) +ia\c,a)}. (1) 

Here, a, b and c are the t2g multiplet given by \a) = \yz), 
\b) = \xz) and |c) = \xy), respectively. The effective 
interaction between sites f and along the 7 bond (see 
Fig. [T]) reads 

H2p - - Jo + JhSp ■ Sp, + JkSJSJ, . (2) 

Jo = ^(15ri + 5r2 + 4r3), J^ = ^(3ri + ra + 2r3), 
J/f = |(ri-r2), where n = 4/([/-3/), = 4/((7-/), 
ra = t^/{U + 21). Here, both Heisenberg and Kitaev 
terms have positive sign, i.e., antiferromagnetic (AF)j^ 
For eg systems in the (111) bilayers, the SOC is ac- 
tivated through the virtual electron excitations to the 
t2g multiplet under the trigonal Ca^ crystalline field i^^i^^ 
Using the basis \a) — jSz^ — r^) and = — 2/^), a 
low-energy Kramers doublet for is given by 

|a) = i={|a,a)+*a|Aa)}, (3) 

where the spin quantization axis is taken along the [111] 
crystallographic axis. For e^, the + sign in Eq. ([3]) is 
replaced by the — sign. This doublet can be gauge 
transformed to --i={|3a;^ — r^,cr) -|- icr|y^ — z^,a)} and 

-^{|3y'^ — r^, cr) -f icr|z^ — x^, a)} with trivial phase fac- 
tors. Thus, the effective interaction is expected to be 
symmetric with respect to the bond direction. Follow- 
ing the same procedure for the t^g systems, the effective 
interaction between sites r and r* is derived as 

Hppi = — Jo + JhSp ■ Spi — JiSpSpi. (4) 

Here, Jo = |(3ri -I- 2r2 + r^), Jh = \{ri + rs), J/ ^ 
i(2ri-r2-r3), where n = 4/([/-3/), r2 = 4/([/-/), 
ra = t'^/{U + I). Now the anisotropic term is described 
as a ferromagnetic (FM) Ising interaction. This comes 
from the fact that the total 5*^ is conserved in the model 
[see Eq. © and Re^ll^. 

The effect of the direct dd transfers, termed ts after 
(ddS) bonding, for the both models is discussed in Ref.l22l. 

Undoped cases: Here we discuss the AF Kitaev- AF 
Heisenberg (AKAH) model for the tag system and the FM 
Ising-AF Heisenberg (FIAH) model for the e^'^ system 
using the parameterization Jh.i = 1 — a and Jk = 2a. 
As Jh and Jk have the same sign, the direct transition is 
expected between the Neel AF at small a and the Kitaev 
SL at large a for the AKAH model. For the FIAH model, 
the planar Neel AF is expected at small a and the FM 
with the spin moment in the [111] direction at large a. 

We now employ the Lanczos exact diagonalization 
for the model Hamiltonians [Eqs. ^ and ^] defined 
on a 24-site cluster with the periodic boundary condi- 
tion. This cluster is compatible with the four-sublattice 




FIG. 2: (Color online) Lanczos exact diagonalization results, 
squared total spins (normalized to its value in the fully polar- 
ized FM state) and the NN spin correlations, for model (a) 
and Bg'^ model (b) obtained on 24-site clusters as a function of 
a. Solid (dashed) lines correspond to original (rotated) spin 
basis. Vertical dash-dot lines are first-order phase boundaries. 
Shaded areas are the parameter ranges for < 31 < U with 
ts = 0. Inset: Controlling parameter a for both tig and e^'^ 
models as a function of 31 /U. Dashed lines include tg — 0.1t,r 
or ts — O.lta- 



transformation^ which changes the original spin S to S. 
Numerical results shown in Fig. [5] (a, b) confirm the above 
considerations. Yet, the SL regime is found to be rather 
narrow for the AKAH model with the critical ac ^ 0.96 
separating it from a magnetically ordered phase. For the 
FIAH model, the phase transition takes place at a = 0.5 
separating the (111) FM phase and the planar Neel AF 
phase. In Ref . |25||26|. the hypothetical Kitaev- Heisenberg 
models with different signs of interactions are studied. 

Natural questions arise where is the "physical" param- 
eter range, i.e., U > 3/, and can t2g systems realize 
the Kitaev SL phase? Now, rewriting Jkj and Jh as 
Jkj = 2Jq; and Jh — J(l — a), respectively, with J the 
normalization, one obtains a = Jkj /{^Jh + Jk,i)- In 
the inset of Fig. [21 we plot a for both the and the 
Cg''^ models as a function of I /U . It is shown that a 
does not exceed 1/5 for the AHFI model and 1/2 for the 
FIAH model; thus both cases fall into the Neel ordered 
regime. The effect of the direct dd transfers is found to 
merely suppress the anisotropic interactions as seen as 
dashed lines. Thus, additional interactions, such as mag- 
netic frustrations, are necessary to realize the Kitaev SL 
phase in t2g systems to suppress Jh- 

Slave-boson mean-field theory: Although the Kitaev SL 
phase is outside the "physical regime" for Mott insulating 
systems, there could emerge novel electronic states by 
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carrier dopingiiiii^ As the two models are reduced to the 
tJ model on the honeycomb lattice at a — >■ 0, one possible 
candidate is the singlet superconductivity (SC) with the 
broken time-reversal (TR) symmetry, so-called d + id^ 
In the opposite limit of the AKAH model, novel SC states 
could be stabilized in connection to the Z2 SL. For the 
FIAH model, on the other hand, the triplet [p) SC states 
may emerge. Here, we examine these possibilities using 
a slave-boson mean-field (SBMF) theory. 

First, we introduce a SBMF method that can be ap- 
plied for Ising-like anisotropic interactions. An 5 = 1/2 
spin operator for a Kramers doublet is described by 
fermionic spinons as = \ f'l!„'T2„, Jfa' with the local 
constraint fl^fra — 1 and being a Pauli matrix. 
Now, a spin quadratic term can be decoupled into several 
different channels as 

SjSj, ^ — — {AppiApp + Xrr'Xrr' + tj^pitpp + 6 ~, cjl-, ) 

+ i (tl* tZi, + el* el, ) , (5) 

' g / J \ ff' TV TV TV ' ' V / 

where A??- = ffjT^^Jp,^, (singlet pairing), tip, = 

fra[iT^^'"]aa' fr'a' (triplet pairing), Xrr' = flafr'a' (spiu- 

conserving exchange term), e~, = fl^T^^Jp,^, (spin- 
non-conserving exchange term). Summation over 7 in 
Eq. ([5]) gives a Heisenberg term. Then, the mean 
field decoupling is introduced to terms having the neg- 
ative coefficient. This recovers the previous mean-field 
schemesi^"— Different decoupling schemes are also used 
in literatures .i^^^i^ The full exp ression of the mean-field 
Hamiltonian is given in Ref. 

We remark on the AF Kitaev limit of the undoped 
tig model. For this limit, we looked for self-consistent 
mean field solutions which respect the underlying lat- 
tice symmetry. Such a solution was found to be given 
by -{x...y.z) = -{<) = {tl) = i{tl) = 0.3812* and 
-(e§) = -(ep = {tD = in) = i{tl) = i{ty) = -0.1188Z 
with the other order parameters and the chemical poten- 
tial being zero. Here, the notation is simplified by re- 
placing the subscript rr' with the bond index p — x,y, z 
connecting the sites f Cz A and r' Cz B: Vp = ~ f. 
It is remarkable that this mean-field solution gives the 
spinon dispersion relation identical to that reported for 
the FM Kitaev model )^^'^° i.e., the ground state of the 
Kitaev model does not depend on the signs of exchange 
constants.^ The current ansatz corresponds to the gauge 
used in Refs.lTll andi30l. and correctly describes a Z2 SL. 

Doping effects: We consider hopping matrices pro- 
jected into neighboring Kramers doublets. In this rep- 
resentation, the hopping matrices are diagonal in the 
isospin index a: Ht = S(r?')cr(4cr'^*"<7 + h.c). The 
hopping amplitude is renormalized according to the rel- 
ative weight of the wave functions as i — ^{t^ + ^ts) 
[^{ta + ts)] for the [e^'^] systems. The double oc- 
cupation is prohibited by the strong repulsive interac- 
tions for c operators. This effect at finite doping can be 
treated by introducing two bosonic auxiliary particles 61^2 



as Cf., => -l^ibl^fp, + abljl-) (Ref.lH) with the SU{2) 

singlet condition Kl ^ iTr FpT^Pl- \Tr t^'bIt^Bp = 0. 

Here, Fp = (^^t "4) ^nd Bp = ~^^) (Ref. 11), 

and the global constraints {K^) = are imposed by 
SU{2) gauge potentials . Doped carriers can be ei- 
ther holes or electrons, and the effect is symmetric for 
our model. We focus on the low-doping regime at zero 
temperature and assume that all bosons are condensed, 
i.e., 5 = Ei.(^IrM ~ Ei/KMP and {Kr&A) = 
{zLi){b^P^B), arriving at the mean-field hopping term: 
i/MF ^ -ftE(.-F>{(T«)/t/rV + h.c.}. Imaginary 
number i arises when the bose condensation inquires the 
sublattice-dependent phase.-- 

Many mean-field parameters have to be solved self con- 
sistently. In order to make the problem tractable, we 
focus on the following five ansatze which respect the six- 
fold rotational symmetry of the underlying lattice as fol- 
lows. First ansatz, termed p SCi, is adiabatically con- 
nected to the mean-field solution in the Kitaev limit given 
above. Here, the relative phase ±« is required between 
the bose condensation at sublattices A and B with the 
SU{2) gauge potentials — = a^J^-^ The second 
ansatz is a p SC, termed p SC2, the third one is a singlet 
SC with the s wave paring, and the fourth one is a singlet 
SC with the d + id pairing. For the latter three ansatze, 
we further introduce the following conditions: 1. Order 
parameters (ej) are assumed to be zero because these 
indeed become zero at large dopings. 2. The bose con- 
densation does not introduce a phase factor. 3. The 
exchange term is symmetric (xp) = (x) and real. Thus, 
these ansatze are regarded as BCS type weak coupling 
SCs. In addition, magnetically ordered states with finite 
(ep = (e^) are considered for the FM Ising case as the 
fifth ansatz. 

Because of the constraint — — a^, the spinon 
density {fl^fp^) differs from the "real" electron density 

(cp^cpcr) in the p SCi phase and a normal phase {{tJ) — 
(ej) — (Ap) = 0) adjacent to it. In many cases, such a 
normal phase has slightly lower energy than the other SC 
ansatze. We discard such a solution as it is an artifact 
by the constraint. 

The schematic phase diagrams for the doped AKAH 
model (a) and FIAH model (b) are shown in Fig. [3] as a 
function of S and a. Here, to see various phases clearly, 
we chose the interaction strength as Jk + Jh — t-^ + ^tg 
and Jj + Jh = tc, + ts. For the AKAH model, the p SCi 
phase is stabilized at a ^ 1 and S ^ 0. Its area is quite 
small as its stability is intimately connected to that of 
the Z2 spin liquid. Large area is covered by the singlet 
SCs, d + id phases at small 5 and s at large 6. This 
behavior results from the fact that the AF Heisenberg 
term dominates the low energy properties. For the FM 
Ising case, the (111) FM phase is stabilized in the large-a 
and small-^ regime. The p SC2 phase is also stabilized 
from the weak coupling mechanism but is found to exist 
only as a metastable phase at larger a and smaller S. 
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FIG. 3: Schematic phase diagrams for the doped AFAH model 
(a) and FIAH model (b) as a function of S and a. Parameters 
are taken as Jk + Jh ='t-w + \ts and J/ + Jh = ta+ts- Phase 
boundaries at finite 5 are the results of the SU{2) SBMF, 
while those at 5 = are results of the exact diagonalization. 
Shaded areas are the parameter ranges for < 31 < U with 
ts — 0. Light lines in (b) are phase boundaries when the FM 
ordering is suppressed. 

The p SCi phase is characterized by the dispersive x° 
Majorana mode and the weakly-dispersive j(^^y^^ modes. 
At finite 5, all modes are gapped by the mixing between 
different Majorana modes due to the finite gauge poten- 
tial a^^y . This results in the finite Chern number -1-1. 

In the (111) FM, spin polarization is 100 % at (5 = as 
in the exact diagonalization result. This large spin po- 
larization persists up to relatively large 5 as carriers can 
move without disturbing the spin ordering. The (111) 
FM area is extended to smaller a at (5 7^ probably be- 
cause the mean-field anzatz for the (111) FM is closer to 
the true ground state at (5 = than that for the d + id; 
it reduces to the true ground state at a > 0.5. Since 
a is reduced by the direct dd transfers, the unconven- 
tional d + id SC is the most probable candidate induced 
by carrier doping. 

Discussion: We now discuss the possible experimen- 
tal realization of our model systems. A (111) bilayer 
of SrIrOa (Ref. HI) would be a good candidate for our 
AKAH model for t|g systems. Also, the FIAH model 
might be realized in a (111) bilayer of palladium oxide 
LaPdOs (Ref.il). This id^t^gcl) electron system con- 
sists of nearly undistorted PdOe octahedra and is ex- 
pected to have the stronger SOC than 3d counterparts 
such as LaNiOa. Carrier doping would be achieved by 



partially substituting Ir by Ru or Os (hole doping) or 
Sr by La (electron doping) for SrIrOa and La by Sr (hole 
doping) or Pd by Ag or Au (electron doping) for LaPdOa. 
It is yet to be clarified whether SrIrOa and LaPdOa are in 
the strong coupling regime, resulting in Mott insulators, 
or in the weak coupling regime, resulting in spin Hall 
insulators or topological metals^i^ Even if these systems 
are in the Mott regime, the Kitaev SL may not be real- 
ized. But, carrier doping would induce novel SC phases 
with d + id symmetry. 

For deriving effective models, the energy hierarchy is 
assumed as U ^ X ^ t. Whether or not such a condi- 
tion is realized in real materials remains to be examined. 
However, the effective transfer intensity is suppressed by 
correlations, and the corresponding hierarchy could be 
achieved self-consistently as discussed in Ref. (111) 
bilayers of perovskite oxides are plausible as the d bands 
are relatively narrow (see for example band structures in 
Ref. El). The form of the NN interaction should not be 
altered even if the above hierarchy is broken as long as 
the local crystal field is maintained and the interactions 
are expressed in terms of S = 1/2 isospins because it 
relies on the symmetry and the spin conservation. 

Realizing Z2 SL and p SC phases may be preferable for 
fault tolerant topological computations. Within the cur- 
rent models, these phases are hard to achieve. For this 
purpose, an alternative route would be looking for sys- 
tems with the FM Heisenberg interaction with which the 
parameter spaces for the Z2 SL in the undoped systems 
and the p SC phases in the doped systems are wideri^ 

To summarize, we studied the properties of Mott insu- 
lators realized in (111) bilayers of TMOs with the strong 
SOC. The low-energy effective models for such insulators 
consist of the anisotropic interaction and the AF Heisen- 
berg interaction. The former is of AF Kitaev-type for 
the systems and FM Ising-type for the e^''^ systems. 
In both the cases, large parameter spaces are character- 
ized by magnetic long range orderings with the narrow 
window for the SL regime in the i^g systems. Yet, car- 
rier doping was found to make the physics of the current 
models more interesting by inducing unconventional SC 
phases in both the cases. The most probable candidate 
is the singlet SC with the d + id, symmetry. In the light 
of the weak SOC limit (Ref. ITTllTsh and the strong cou- 
pling limit (Ref. [H), TMO (111) bilayers would provide 
even richer quantum behavior as a function of Coulomb 
interactions, the SOC and carrier doping. 
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ment of Energy, Basic Energy Sciences, Materials Sci- 
ences and Engineering Division. 
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Supplementary material 



SI. Model Hamiltonian 



In (111) bilayers, the SOC in the eg multiplet is acti- 
vated through the virtual electron excitation to the t2g 
multiplet under the trigonal Csv crystalline field. The 
resulting SOC is expressed as 



by 



Sl.l. Transfer matrices 



For both t2g and Cg systems, the hopping term is given 



i^rr') oo' a 



(SI) 



so 



E 



dl 



V 

T ,r 

recr ee' ' 



(S8) 



where f are Pauli matrices. Here, the spin quantization 
axis is taken along the [111] crystallographic axis. By 
diagonalizing the Hamiltonian Eq. (jSSp . one obtains the 
Kramers doublet given by Eq. (3). 



where dt^^ is the creation operator of an electron at site 
r, orbital o with spin a. The hopping amplitude t~, is 
determined from the Slater-Koster formula^^ with oxygen 
p orbitals located between sites r and . 

In t2g systems, electrons hop from site to site through 
TT bonding (pdir) between the neighboring t2g orbitals and 
oxygen 2p orbitals in between and through weak direct 
overlap {dd6). The dependence of the NN transfer ma- 
trices on the orbital and direction is given as follows: 



r,f±y{z) r,r±£(a;) 
j-aa j.bb 



J.CC 



J.CC 



(S2) 
(S3) 



with the use of the following convention for the or- 
bital index: \a) = \yz), \b) = \zx), and |c) = \xy). 

Ri (pdir)'^ /A with A the level difference between TM 
d orbitals and O p orbitals, and tg = (dd6). 

In eg systems, electrons hop from site to site through 
a bonding (pda) between the neighboring Cg orbitals 
and oxygen 2p orbitals and and, similar to t2g systems, 
through weak direct overlap (dd5). The dependence of 
the NN transfer matrices on the orbital and direction is 
given as follows: 



4-aa 
r,f±z r^f±z 

r,r±z r,r±z 



j.ee 



1 

4 

''r,r±y ~^ 



to- -|- its 





'ta 0' 






ts 






-V3{ta 


-ts) 



it„ 



ts 



tc + its \/3(tcr — ts) 
Vi{tcr ~ ts) 3tcr + ts 



(S4) 

(S5) 
(S6) 



with the basis \a) = |3z^ — r^) and 
t„ « {pda)yA. 

SI. 2. Spin-orbit coupling 

The SOC for the t2g model is given by 
A 



so 



(S7) 



raa rr r ' 
ei2g 



where the Levi-Civita antisymmetric tensor. 



SI. 3. Local Coulomb interactions 

For simplicity, we neglect the coupling between t2g 
electrons and eg electrons in the local interaction. Thus, 
the multiorbital interaction for both the cases can be ex- 
pressed as 



OO 

o" o" 



E 

crcr' 



'dl. 



roa^to'a'^^o 



where the orbital indeces o, . . . o'" run through either 
the t2g multiplet or the eg multiplet (see for example 
Ref. SI). Because of the orbital symmetry, a well know 
relation U = U' + 21 holds, where U = U°°°° (intraor- 
bital Coulomb), U' = U°°'°°' (interorbital Coulomb), 
/ = jjoo'o'o (interorbital exchange) = u°°°'°' (interor- 
bital pair transfer) for o ^ o' ^ and other components are 
absent. 

Equation (jS9p can be easily diagonalized when two 
electrons occupy site r. Resulting energy eigenstates and 
eigenvalues are as follows: 



for ilg^ 



'T2 



and 



^2 



U — 31 (nine-fold degenerate) 

U ~ I (three-fold degenerate) 

U ~ I (two-fold degenerate) 

U + 21 (non degenerate) 



U — 31 (three-fold degenerate) 
U — I (two-fold degenerate) 
U + I (non degenerate) 



(SIO) 



(Sll) 



for Cg. These energy levels determine the excitation en- 
ergy for tlgt^g ^ tlgtlg 0x1^ e^^ e^ ^ e^e°, and also 



symmetry. 



t%„t?,„ ^ t^„t%„ and BgCg ^ CqCg by the particle-hole 



SI. 4. Effective interactions 

Considering the hmit U ^ A(A) ^ i7r(cr) and U » 
(trigonal crystal field splitting for t2g multiplet), we in- 
clude the SOC in the initial states and the final states 
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of the second-order perturbation with respect to Ht , ar- 
riving at the effective Hamiltonian Eq. (2) for systems 
and Eq. (4) for ej'^ systems. 

Here we provide full expressions for the effective inter- 
actions for systems [Eq. (2)] and Cg'^ systems [Eq. (4)]. 
For systems, we obtain 

Jj, = ^{l-2u + u^){ri-r2), (S12) 

Jh = ^{(3 + 6i/)ri + {l-2u + z/2)r2 + {2 + 2u + ly^s}, 

(S13) 

withri = tl/iU-3I), r2 = tl/{U-I), = tl/{U + 2I), 
and u = ts/tT^. For e^'^ systems, we obtain 

Ji = i{(2 - 41/ + 2i/2)r-i - (1 - 41/ + iy^)r2 - (1 + iy^)r3}, 

(S14) 

JH = ^{l + 2u + i^^){n+r3), (S15) 

with n = tl/{U-3I), r2 = tl/{U-I), rs = tl/{U + I), 
and u = ts/ta- 

We check these interactions by considering two limit- 
ing cases, (i) I/U — )• 0, both Jk and J/ become zero. 



This is because, in the intermediate states of the second- 
order perturbation processes, interorbital contributions, 
sum of ^T2 and ^Ti for 4g and sum of ^A2 and for 
e],'^; vanish and only intraorbital contributions remain. 
Intraorbital contributions involve configurations such as 
\a^ai), resulting in the AF interactions Jh- (ii) ts — >■ 
or ts — tcr, the directionality comming from d-orbital 
wave functions is lost. Thus, in this case, Jk vanishes 
for the t2g model. On the other hand, J/ remains finite 

for the Cg'^ model. This is because the total S \\ [111] is 
conserved. 



S2. Mean field Hamiltonians 

After the mean-field decoupling, the single-particle 
Hamiltonian for the AF Kitaev-AF Heisenberg model for 
t2g systems is expressed as 

hmf^j^y:^ ^imn.' + (S16) 

Here, a Nambu representation is used with 4-component 
spinors 4^ given by = {fl^^, 4^,^, Lua^-kBa) ^ 
and an 8 X 8 matrix H given by 



H{k) 



-a'^ Sera' Xcrcr'{k) (a^ + )£aa' 



(S17) 



£-|-4, = —£4,-1- = 1 is the antisymmetric tensor. x{k) and A(fc) are 2x2 matrices given by 

p p 



(S18) 
(S19) 



respectively, with f° being the 2x2 unit matrix, fp is a unit vector connecting the nearest-neighboring sites along the 
p bond, i.e., — i~V3/2, —1/2), fy = (•\/3/2, —1/2) and fz = (0, 1). The prefactor for i comes from the mean-field 
decoupling for the bosonic term {bAib^Bi ~ ^a2^b2)- For the p SCi phase, the bose condensation at one of the two 

sublattices acquires phase i, thus we have {bAib^Bi ^ ^A2^B2) ~ + (^2)^} = ±iS, while for the other phases 

considered, only bi bosons condense at finite 6, thus {bAib^i — ^^2^52) ~ (^1)^ = S- Hq is a constant term given by 



P\\2\ 



3 

8 ^p 



Ep^i/(l(Xp)P + l(Ap)|2). 



For the FM Ising-AF Heisenberg model for ej'^ systems, including the uniform magnetic momrnt m = {fj^fif 
filfa), we have 



Hik) 



xAik) 



Xcrcr' (k) 
-'aa' 4 * 



-A„,,{-k) 



{a^ + iay)£a 



-KA-k) a^^^a' + lJimol^. -X.t^A-k) 



(a^ - iay)ea'o- 



(S20) 
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with 



xW = -jE^*''''[{4'^W* + 3JH(x;)}T°-JH(er)r1-jE E e^'-'^Mellr^ (S21) 



(S22) 



and the constant term is given hy = \ E^=x,^ + + i JH(3|(Xp)P-|(eg)P+3|(A,)P). 



S3. Majorana representation for the p SCi phase in Spin operators are represented by these Majorana 



the AF Kitaev-AF Heisenberg model 

For the p SCi phase, a mean field solution which 
respect the underlying lattice symmetry is given by 
{X.,y,z) = -iA', -(el) = (t-) = = I A and 

-kO = -{el) = m) = {t%) = i{ty) = z{ty) = zB, where 
A,A',B are real numbers. Spinon operators can be rep- 
resented as linear combinations of A/[ajorana fermions. 
Here, we use the gauge given by You et al.:^^ 



fermions as S'^ = ix°X^) with the local constraint 
x'^X^X^X^ = 1/4 and the normalization {x'^, x'^ } = ^77'- 





■^{x°+ixn, 


(S23) 


/;= 




(S24) 


h = 


^(ix^-x^), 


(S25) 




^Hx-x^). 


(S26) 



J 



By inserting the mean field parameters and spinon operators in terms of Majorana fermion operators, we obtain 
the following expressions for the mean-field Hamiltonian along the x, y, and z directions: 



1 



1 



= -8{4i*<5 + {Jk + 3Jh)iA'}{x'aX% + xlxl + x\x'b + X^xIj) - g^if*^(x^X^ + xlx^ - x'aX'b " X^xl.), 

(S27) 



E. 



MF 



-{4US + {Jk + SJh)iA'}{x°aX% + xlxl + x'axI + xlxh) " 8^x*^(x^X°b " X^X^ + X^X^ - xlx%), 

(S28) 



= - o {4ti<5 + {Jk + SJH)iA'}{x°AX% + x'kxh + X^X^ + X^x'b) - ^JKiA{x\x% - X^X^ - X^X^ + xlxl. 



(S29) 



In the undoped Kitaev limit ((5 = and Jh = 0), 



At finite S, the gauge potentials have to be explicitly 



A = A' and, therefore, this mean field Hamiltonian is included as acts as the chemical potential for spinons. 



reduced to 



The local term involving the gauge potentials is given in 



H. 



^^ = -^jKa(xVs + xM- 



(S30) 



Thus, we recover the correct Kitaev limit with the disper- 
sive x° raode and the dispersionless x'^'^'^ modes. With 
finite Jh, A' ^ A, and the x^'^'^ modes acquire finite 
dispersions. 
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terms of spinons or Majorana fermions as 

- \a^{-^x\^ - *X'^X^ - b\h2 - bib,) 

+ ^«' Hx\' - ^X^X" - b{b, + blb^) . (S31) 

Thus, in order to have the correct lattice symmetry (a; ^ 
y O z) at finite doping, all gauge potentials must have 
the equal absolute value. At the same time, the mixing 
between different Majorana modes generates excitation 
gaps, resulting in the finite Chern number. -'^^ 

Dispersion relations of the Majorana fermions are pre- 
sented in Fig. lSll for various choices of parameters. In the 



undoped Kitaev model (a), only the gapless x" niode is 
dispersive. With finite doping 6 (b), x^'^'^ modes become 
dispersive and the x" niode is gapped. With finite Jh 
(c), ^^'V'^ modes become dispersive while the x*^ mode 
remains gapless. With Jk — tj^^Jn — and 5 — 0.03, 
the gap amplitude is ^ 3 x lO^^iTr and, therefore, invis- 
ible in this scale. Softening of the x^'"'^ modes is not 
strong enough to close a gap. This results in the Chern 
number -t-1. 
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FIG. SI: Dispersion relations of the Majorana ferrnions for 
the AF Kitaev-AF Heisenberg model, (a) Undoped Kitaev 
limit, (b) doped Kitaev, (c) undoped Kitaev-Heisenberg. Pa- 
rameter values are indicated. 



